Skip to content

Main gets work time without the audio thread wait, benchmark tools that skip diagnostic files and the real menu names in the docs - #38

Merged
redcode9 merged 10 commits into
features/state-of-the-art-development/mainfrom
features/state-of-the-art-development/wip/perf-method
Nov 16, 2026
Merged

redcode9 merged 10 commits into
features/state-of-the-art-development/mainfrom
features/state-of-the-art-development/wip/perf-method

Conversation

@redcode9

Copy link
Copy Markdown
Owner

Review-fix round for wip/perf-method. The branch was fast-forwarded to main 391f5e9 first, so this PR holds nine commits after it: one PSP code change in source/src/sound.c with its perf_metrics change and host test, two script fixes with their tests, and six docs commits. The report format does not change: no key is added or removed.

What was wrong and what changed:

  • Work time counted the sound sync's sleep as work (25cd5a1, note in c092e2a). With manual or no frameskip, synchronize_sound() sleeps in 100 us steps until the audio thread takes a block. That sleep falls inside the frame time, but only recorded waits are taken out of it, and this one was never recorded. So in paced captures with manual or no frameskip, work-time p50/p95/p99, frames-work-over-budget and phase-main-update-gba-us held the audio wait, and an audio pacing difference could look like a CPU regression in compare-benchmarks' work-p95 gate. Captures with auto frameskip and the unbound benchmark were not affected.
    • perf_metrics.h gets a last wait site, PERF_METRICS_WAIT_AUDIO_DRAIN. perf_metrics_record_wait adds it only to the frame's waits: no VBlank count or total, and the backpressure flag is untouched, so late frames do not change.
    • synchronize_sound() sets a flag before each sleep. After the loop, while a capture runs, it records the time since started under the new site. started is taken after the auto frameskip's VBlank wait, so nothing is counted twice, and a drain right after that wait is covered too.
    • This costs one sceKernelGetSystemTimeWide read in sound.c, only during a capture and only on frames where the loop slept. With no capture running, the game pays only one extra check, on frames where the loop slept.
    • docs/PERFORMANCE.md now lists the sound sync's wait for the audio thread among the waits that phase-main-update-gba-us leaves out.
    • tests/test_perf_metrics.c gets test_audio_drain_leaves_work_only: work is the frame minus the audio VBlank wait, the drain and the GE wait; only the VBlank wait is counted at its site; backpressure and present counts stay 0; the frame still counts as late.
  • compare-benchmarks failed on the -alloc.txt files every diagnostic build writes beside its reports (e421190). main_perf_write_alloc_trace writes one next to every report in a diagnostic build, and the padding experiment (PRF-10.1) needs that profile. collect_groups skipped only .md and -core.txt, so --baseline-dir / --candidate-dir on a copied diagnostic benchmarks/ folder stopped with "benchmark is not stopped", and the experiment could give no verdict. A new NOT_REPORTS constant (.md, -core.txt, -alloc.txt) is what collect_groups now skips. release_gate.py reads groups through collect_groups, so it gets the same fix. tests/test_compare_benchmarks.py writes a run-0-alloc.txt in the real format beside the -core.txt case and checks the group still has 3 reports.
  • perf-table.py did not skip what compare-benchmarks skips (692c272). perf-table.collect skipped only .md, so a -core.txt or -alloc.txt beside an archived report made perf-table.py --update fail with "invalid report line", while check-tree.sh and compare-benchmarks accepted the same archive. It now skips compare_benchmarks.NOT_REPORTS. test_empty_archive in tests/test_perf_table.py also writes run-1-core.txt and run-1-alloc.txt and still gets "No reports archived yet.".
  • The docs sent testers to "Diagnostics > Start measurement", a row that does not exist (b19e4fe). The rows are "Start recording", "Stop recording" and "Save report" (message_eng.h:1160-1164). benchmarks/README.md, docs/PERFORMANCE.md, README.md, the perf-baseline-5 steps in docs/hardware-tests.md and a comment in tests/test_perf_video_sentinel.c now use those names. perf-baseline-5 gives the steps for the reports the stable release gate compares. The README.md and docs/hardware-tests.md lines were older than this branch. No tracked file says "Start measurement" any more.
  • The changelog called the menu "Link" (b19e4fe, db52e1a). The category is "Link Cable" (MSG_CATEGORY_LINK_CABLE). "Link > Restore pre-link save" and "the Link menu says why", both in the Unreleased save-store items, now say Link Cable, rewrapped to 76 columns.
  • CONTRIBUTING.md's list of what the diagnostic profile adds left out the two Unbound benchmark rows (bc79c4f). build_profile.h switches them on with ENABLE_DIAGNOSTIC_BENCH_ARMS, and they also shift the Diagnostics row numbers. The list now names the rows with rendering off and with audio off.
  • PRF-3.2 and PRF-3.3 were marked «codice pronto», but no build can check their acceptance (cfc59fe, c092e2a). Both ask for the same state, checked with the F-120 hash. In main the bench hashes nothing: input_diagnostic_trace_begin_bench in input.c frees the replay hashes, the perf report has no hash key, and the game goes back to paced play when the bench ends, so the end state cannot be captured afterwards. A hardware session on these tickets would have wasted console time. Both notes now read «in parte» and say:
    • main's bench computes no state hash;
    • F-120's hash code (ENG-3.3, 91bc1ef) is in main since 2ff83ab, but ENG-3.3 waits for its PSP proof;
    • the end-of-bench hash (one at the last frame of a trace bench, without the audio region, written in the perf report as bench-state-hash) is commit bcf3657 on wip/replay-oracle and reaches main with that branch's review-fix PR;
    • for PRF-3.2, the arms are then compared with the same trace and slot, audio-off included, because the hash leaves out the samples that the audio-off arm does not write.
      Round 1 wrote that F-120 was in main and that the hash would come with the replay-oracle PR before any branch had the code. c092e2a corrects the F-120 wording, and bcf3657 now exists.
  • The ticket notes still waited on work that is in main (cfc59fe). The header and PRF-1.5's «aspetta» listed PRF-4.1 (wip/platform), in main since cf5ef4f. PRF-1.5 now waits only for CMP-1.4, with «PRF-4.1 è in main (cf5ef4f)». ENG-3.2 for PRF-1.7, in the header and on PRF-1.7's line, now says it is in main since 2ff83ab but not yet proven on PSP. A second finding reported the same PRF-4.1 wait, and cfc59fe fixes it too.
  • The unbound benchmark can push Exact off the Media Engine without the report showing it (cfc59fe, c092e2a, fb7ec68). During the bench nothing waits. If the Media Engine falls behind for 300 frames in a row (video_me.c), Exact moves to the main CPU until the EBOOT restarts, and the report writes only renderer. Whether this ever happens depends on PSP timing nobody has measured, so the code is unchanged.
    • PRF-3.1's note asks its hardware session for one more bench with Exact on the Media Engine, on one light title and one heavy title. It says how to read the result: start the EBOOT, open the game and launch the bench straight away (the bench loads the slot itself), quit right after it, and decode the runtime log with python3 tools/decode_runtime_log.py. Look for exact-renderer=cpu-too-slow instead of media-engine, and for late frames in the second value of me-frames/frames-late/me-busy-us-per-frame (LINK_COUNTERS groups 11 and 9, written at quit and counted from the EBOOT start, so group 9 then holds almost only the bench's frames).
    • docs/PERFORMANCE.md's "every frame is drawn" now adds that with Exact on the Media Engine, a frame the Media Engine has not finished in time shows the previous one again and counts as late.

Tickets whose status changed:

  • PRF-3.2 and PRF-3.3: from «codice pronto, manca la prova su PSP» to «in parte». They need bench-state-hash in main, then their PSP run. Checkboxes stay empty.
  • PRF-1.5: its «aspetta» lists only CMP-1.4 (wip/open-bios). It also still waits for PRF-2.3 (decision (m)) and a hardware session. Checkbox stays empty.
  • PRF-1.7: its «aspetta» on ENG-3.2 stays, now saying that ENG-3.2 is in main since 2ff83ab but not proven on PSP. Checkbox stays empty.
  • PRF-2.1 stays ticked. Its note records the sound sync fix and the extra clock read, and says the PSP build of the changed sound.c shows in this PR's CI. Its acceptance limits clock reads only in main.c and psp_video.c, and check-performance.sh still passes.
  • PRF-3.1 and PRF-10.1 keep their status. PRF-3.1's note gains the Exact on the Media Engine run, and PRF-10.1's says the folder comparison skips the -alloc.txt files since e421190.
  • No ticket was ticked or unticked, and docs/BACKLOG.md does not change.

Tests on this host (clang with the freestanding host shim, UBSan in trap mode):

  • test_perf_metrics, test_perf_monitor and test_perf_report, built with -std=c99 -Wall -Wextra -Werror: pass, plain and with UBSan. The fixer ran them after 25cd5a1 and at cfc59fe, and reran test_perf_metrics at db52e1a. The reviewer ran all three from a git archive copy in both rounds.
  • Mutation checks on the new perf_metrics.c case: dropping the case, counting the drain as a VBlank audio wait, setting the backpressure flag and adding nothing. test_audio_drain_leaves_work_only catches all four, plain and with UBSan. Before this PR no test covered the drain.
  • python -m unittest of test_compare_benchmarks.py, test_perf_table.py, test_release_gate.py and test_hardware_session.py at cfc59fe: 102 tests OK. One test was skipped because it needs zip and unzip. At db52e1a, test_compare_benchmarks.py and test_perf_table.py: 49 tests OK.
  • Five mutants of the skip lists: compare-benchmarks without -alloc.txt and without -core.txt, and perf-table back to .md only, without -core.txt and without -alloc.txt. The tests catch all five.
  • python scripts/hardware-session.py --check: passes, with 29 tests in the catalogue and 18 valid result rows.
  • From CR-stripped copies with PYTHON=python: check-tree.sh passes. It covers messages, menu layout, lifecycle, video, performance, script syntax, forbidden names, module tests and the source tree. check-performance.sh, check-scripts.sh, check-messages.sh, check-menu-layout.sh, check-lifecycle.sh and check-video.sh pass on their own too.
  • The reviewer cut synchronize_sound() out and compiled it against stubs with the PSP warning flags plus -Wextra -Wmissing-prototypes -Werror. It compiles cleanly. This is not a psp-gcc build.
  • git diff --check 391f5e9..HEAD is clean. Every changed file is LF, no message table changed, the forbidden-name check passes, and the commit messages carry no attribution lines.
  • git merge-tree of HEAD with every wip/ branch based on 391f5e9 merges cleanly, except wip/release-and-ci (see below).
  • No Makefile change is needed. The new C case is in test_perf_metrics.c, which is already in HOST_TESTS and HOST_SANITIZE_TESTS. The Python files are already on the unittest line of make check.
  • CI will run make check, make test-sanitize, make check-cbmc and both PSP builds on this PR.

Not verified here:

  • The PSP build of the changed sound.c, which is built with -Werror. This machine has no PSPDev. Fixer and reviewer each read the change: the flag is declared and used, the enum comes in through perf_monitor.h, and the u64 difference is cast to u32 as in the code around it. CI has to confirm it.
  • make check, make test-sanitize with gcc and ASan, and make check-cbmc: this machine has no make or gcc.
  • tests/test_perf_video_sentinel.c was not rebuilt, because it needs the PSP SDK headers. Only a comment changed there.
  • The effect of the drain fix on real paced captures with manual or no frameskip has not been measured on a PSP.
  • Whether the Media Engine falls behind during the unbound benchmark needs the PRF-3.1 hardware run.
  • bcf3657 is only on the local wip/replay-oracle branch. This round did not build it or run its tests. The PRF-3.2 and PRF-3.3 notes assume it reaches main with that branch's review-fix PR.
  • No hardware run. CI has not run on these commits, because nothing was pushed from here.

Files of other branches touched (rule 5):

  • source/src/sound.c (wip/audio): in synchronize_sound(), one local flag and one guarded perf_monitor_record_wait after the drain loop.
  • docs/hardware-tests.md, the shared hardware test catalogue that wip/release-and-ci started: one word in the perf-baseline-5 steps.
  • CHANGELOG.md (shared): the two save-store Unreleased lines about the pre-link copy, rewrapped. wip/save-store's review-fix PR (a59a3e4) makes the same change with the same text, so the two merge cleanly in either order.
  • CONTRIBUTING.md (shared): the diagnostic profile list. wip/release-and-ci's review-fix branch (0c1440e) adds the same two benchmark rows to the same sentence, in other words, and git merge-tree reports a conflict there. Whichever PR merges second keeps one mention of the rows.
  • README.md (wip/docs): the Performance comparison steps.
  • docs/PERFORMANCE.md, benchmarks/README.md and tests/test_perf_video_sentinel.c are this branch's own. wip/replay-oracle and wip/ui-foundation also change docs/PERFORMANCE.md in their review-fix branches, and both merge cleanly with this one.

Skipped, with the reason:

  • The code changes the Media Engine finding suggested: keeping the bench's late frames from the fallback count, or waiting for the Media Engine frame, and writing the Media Engine state and late-frame count in the report. The finding is uncertain until the PRF-3.1 run shows whether the Media Engine falls behind, and the fix instructions asked for no code change.
  • The end-of-bench hash itself is not in this PR. It belongs with the F-120 hashes on wip/replay-oracle (bcf3657).
  • Four nits from the round 2 review, which found no blockers or minors, so the round closed without them. Two of them are for whoever merges:
    • To check before merging: CI is green on both PSP profiles. PRF-2.1 stays ticked while the changed sound.c has never been built for the PSP. If either profile fails on sound.c, untick PRF-2.1 and mark it «in parte» until it is fixed.
    • To check before merging: the owner confirms two statements the notes make. PRF-2.1's note says «il proprietario conferma» that its «Cosa» («usa solo tempi già misurati») allows the extra clock read. The PRF-3.2 and PRF-3.3 notes cite a «decisione del 2026-11-16» for the end-of-bench hash. Both came with the fix instructions, and nothing in the repo records them. If either is not confirmed, PRF-2.1 should say «il proprietario decide se», and the date should go from PRF-3.2 and PRF-3.3.
    • PRF-3.3's note describes bench-state-hash without «senza la regione audio», which PRF-3.2's note gives. The hash leaves out the audio samples, so it is weaker than a full F-120 hash, and whoever ticks PRF-3.3 should know. The fix is to add those words to PRF-3.3's parenthetical.
    • fb7ec68 reflowed the Unbound benchmark paragraph of docs/PERFORMANCE.md only as far as "waits for audio. The game gets no input. The capture then", which leaves a 57-column line before the old lines. It renders the same in Markdown.
  • No CHANGELOG.md entry for the work-time fix: work time is new since 1.4.3, and its Unreleased entry already says "work time without the waits".

The branch stays open, so docs/wip/perf-method.md stays.

With frameskip set to manual or none, synchronize_sound sleeps until the
audio thread takes a block, and nothing recorded that sleep, so a paced
capture counted it in work-time, frames-work-over-budget and
phase-main-update-gba-us. The sleep now goes into the frame's waits as
its own site, which no VBlank wait counter shows.
Every diagnostic build writes perf-<time>-alloc.txt beside each report,
so --baseline-dir and --candidate-dir on a diagnostic build's folder,
as the padding experiment asks, stopped on that file as a report that
is not stopped. The files that are no report are now one list.
A -core.txt or -alloc.txt beside an archived report passed check-tree
and compare-benchmarks but stopped perf-table.py --update as a report
that is not stopped. perf-table now reads the same list of files that
are no report.
The measurement steps sent testers to Diagnostics > Start measurement,
a row that does not exist: it is Start recording, then Stop recording
and Save report. The changelog's restore item is under Link Cable.
PRF-4.1 and the F-120 hash code are in main, so PRF-1.5 waits only on
CMP-1.4. PRF-3.2 and PRF-3.3 are partial, not code complete: the bench
hashes nothing, so their same-state acceptance waits for the end of
bench hash of the replay-oracle review fix. PRF-3.1's hardware run also
checks that the bench does not switch the Media Engine off, and the
notes of PRF-2.1 and PRF-10.1 name this round's fixes.
The PRF-3.2 and PRF-3.3 notes now say that only F-120's hash code is in
main, with ENG-3.3 still waiting for the PSP, and that the end-of-bench
hash is written in wip/replay-oracle. The PRF-3.1 note says to start the
benchmark right after the EBOOT starts, since the quit counters count
from there, and the PRF-2.1 note names the clock read sound.c now takes.
CONTRIBUTING.md: both sides listed the two Unbound benchmark rows of
the diagnostic profile. Keep this branch's wording, which follows the
real row names, and main's pointer to docs/PERFORMANCE.md.
@redcode9
redcode9 merged commit 247c8f2 into features/state-of-the-art-development/main Nov 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant